r.clump: Fail with the clump limit instead of overflowing the index - #7853
r.clump: Fail with the clump limit instead of overflowing the index#7853Pranav-error wants to merge 2 commits into
Conversation
r.clump asked G_realloc() for 18446744065119617024 bytes and stopped with a memory error, which reads as a memory problem when it is not one. label and nalloc are both 32 bit. nalloc starts at INCR and grows in INCR steps, so it lands exactly on 2^31 and wraps to INT_MIN, and nalloc * sizeof(CELL) then converts to (size_t)(INT_MIN * 4), which is 18446744065119617024 - the number in the report. Clump IDs are raster values of type CELL, so as @metzm said on the issue the limit of one CELL cannot be raised. Report it as the limit it is instead, and stop INCR short of INT_MAX so nalloc cannot overflow either. Fixes OSGeo#6412
| /* start a new clump */ | ||
| if (label >= MAX_LABEL) | ||
| G_fatal_error( | ||
| _("Too many clumps: the maximum number of clumps " |
There was a problem hiding this comment.
| _("Too many clumps: the maximum number of clumps " | |
| _("Too many clumps: the maximum supported number of clumps " |
This change might need reformatting.
| /* start a new clump */ | ||
| if (label >= MAX_LABEL) | ||
| G_fatal_error( | ||
| _("Too many clumps: the maximum number of clumps " |
There was a problem hiding this comment.
| _("Too many clumps: the maximum number of clumps " | |
| _("Too many clumps: the maximum supported number of clumps " |
metzm
left a comment
There was a problem hiding this comment.
A small suggested change to the error message, otherwise this is fine, thanks!
Per review: say "the maximum supported number of clumps". Re-wrapped the string so the lines stay within 80 columns.
|
Thanks — applied your wording in both places. I re-wrapped the string so it stays within 80 columns, so the line breaks fall differently from the suggestion but the text is yours verbatim: This is ready on my side, but I cannot take it out of draft: new contributors here are limited to five open non-draft PRs and I am at that limit, so |
Fixes #6412.
r.clumpstopped withwhich reads as a memory problem. It is not one, and the reporter went looking for a disk-based mode because of it.
Where the number comes from
labelis aCELLandnallocanint, both 32 bit:nallocstarts atINCRand only ever grows byINCR, so it is always a multiple of 1024 and lands exactly on 2³¹, which as a signed 32 bit value isINT_MIN.nalloc * sizeof(CELL)then convertsINT_MIN * 4tosize_t:That is the number in the report, so this is the path taken rather than a guess.
labelitself overflows one step earlier, which is undefined behaviour and would give a negative clump ID.The fix
@metzm said on the issue that the real problem is the clump count, and that the limit "can not be raised" — clump IDs are the raster values, so the ceiling is one
CELL. So this reports the limit instead of overflowing past it, at both places that start a new clump.MAX_LABELstopsINCRshort ofINT_MAXrather than at it, sonalloc— which moves inINCRsteps and must exceedlabel— cannot overflow either. The cost is 1024 of 2.1 billion possible clumps.The new message says what the limit is and what to do about it, since the previous one sent the reporter after the wrong problem.
Verification
I cannot add a regression test for this: reaching it needs a raster with more than 2³¹ clumps, so the reporter's 220000 × 240000 region is roughly the smallest case. Rather than assert it untested, the mechanism above is arithmetic that anyone can check against the reported number.
What I did check: builds clean,
clang-formatreports no changes, and the four existing tests inraster/r.clump/tests/test_clump.pystill pass.